Skip to content

WIP: Test CI#14

Closed
mprahl wants to merge 2 commits into
opendatahub-io:orgnization-supportfrom
mprahl:multi-tenancy-backend-only
Closed

WIP: Test CI#14
mprahl wants to merge 2 commits into
opendatahub-io:orgnization-supportfrom
mprahl:multi-tenancy-backend-only

Conversation

@mprahl

@mprahl mprahl commented Nov 21, 2025

Copy link
Copy Markdown

Summary by CodeRabbit

Release Notes

  • New Features

    • Added multi-tenant workspace support for logical isolation across teams/projects within a single MLflow server
    • Introduced workspace management APIs: create, list, get, update, and delete workspaces
    • Added workspace-scoped permissions with READ, EDIT, and MANAGE access levels
    • Workspace artifacts and resources are now isolated and scoped to their workspace
    • Available via Python client, REST API, and CLI
  • Configuration

    • Added --enable-workspaces server flag and MLFLOW_ENABLE_WORKSPACES environment variable
    • Workspace support requires SQL database backend (opt-in feature)
  • Documentation

    • Added comprehensive self-hosting guides covering architecture, configuration, permissions, and workspace providers

✏️ Tip: You can customize this high-level summary in your review settings.

- Introduce the workspace store abstraction (SQLAlchemy + REST), name validation, and registry/URI utilities with CLI flags and env vars.
- Expose workspace CRUD endpoints in the Flask server with proper guards/feature flag wiring plus MlflowClient surface methods.
- Add the Workspace entity, tests for the stores/registry/client, and docs entries so the new multi-tenant provider can be exercised end-to-end.

Signed-off-by: mprahl <mprahl@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Nov 21, 2025

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

This pull request introduces comprehensive multi-tenancy workspace support to MLflow, enabling logical isolation of experiments, models, and permissions within a single server. New workspace store implementations, CRUD APIs, database schema, server routing, permission management, and workspace context management are added alongside extensive test coverage and documentation.

Changes

Cohort / File(s) Summary
Workspace Core Entities
mlflow/entities/workspace.py, mlflow/store/workspace/*
New Workspace entity with to_dict method; AbstractStore interface for workspace providers; SqlAlchemyStore and RestWorkspaceStore implementations with full CRUD; workspace name validation (Kubernetes-like rules).
Workspace Client & Registry
mlflow/tracking/_workspace/client.py, mlflow/tracking/_workspace/registry.py, mlflow/tracking/_workspace/utils.py, mlflow/tracking/_workspace/context.py, mlflow/tracking/_workspace/fluent.py, mlflow/tracking/_workspace/__init__.py
WorkspaceProviderClient for CRUD operations; WorkspaceStoreRegistry with pluggable provider discovery; context manager for workspace scoping; fluent API (set_workspace, list_workspaces, etc.); URI resolution utilities.
Workspace Properties on Existing Entities
mlflow/entities/experiment.py, mlflow/entities/model_registry/registered_model.py, mlflow/entities/model_registry/model_version.py, mlflow/entities/evaluation_dataset.py, mlflow/entities/webhook.py
Added optional workspace parameter to constructors; workspace property exposure; workspace propagation in to_dict/from_dict conversions.
Database Schema & Migrations
mlflow/server/auth/db/migrations/versions/2ed73881770d_workspace_permissions.py, mlflow/server/auth/db/models.py, mlflow/store/workspace/dbmodels/models.py, mlflow/store/db/utils.py
New workspace_permissions table with composite primary key; updated registered_model_permissions with workspace column and unique constraint; SqlWorkspace and SqlWorkspacePermission models; migration upgrade/downgrade paths.
Authentication & Permissions
mlflow/server/auth/__init__.py, mlflow/server/auth/client.py, mlflow/server/auth/config.py, mlflow/server/auth/entities.py, mlflow/server/auth/routes.py, mlflow/server/auth/sqlalchemy_store.py, mlflow/server/auth/basic_auth.ini
AuthContext for username/admin tracking; workspace permission CRUD methods; list/set/delete workspace permissions endpoints; WorkspacePermission entity; grant_default_workspace_access config flag; workspace-scoped permission queries.
Server Integration & Routing
mlflow/server/__init__.py, mlflow/server/handlers.py, mlflow/server/otel_api.py, mlflow/server/workspace_helpers.py
Workspace request lifecycle handlers (before_request, teardown); workspace endpoints (list/create/get/update/delete); workspace-scoped artifact paths; RBAC resource type resolution; OTLP workspace context handling.
Tracking & Registry Workspace Awareness
mlflow/store/tracking/abstract_store.py, mlflow/store/tracking/sqlalchemy_store.py, mlflow/store/tracking/rest_store.py, mlflow/store/model_registry/abstract_store.py, mlflow/store/model_registry/sqlalchemy_store.py, mlflow/store/model_registry/base_rest_store.py, mlflow/store/tracking/dbmodels/models.py, mlflow/store/model_registry/dbmodels/models.py
supports_workspaces() method added to AbstractStore subclasses; workspace-aware query filtering; experiment/run/dataset/model scoping to active workspace; artifact root resolution per workspace.
Workspace Mixin & Artifact Repository
mlflow/store/workspace_aware_mixin.py, mlflow/store/artifact/artifact_repo.py, mlflow/store/artifact/artifact_repository_registry.py, mlflow/store/artifact/http_artifact_repo.py
WorkspaceAwareMixin for session/workspace context management; for_workspace() method on ArtifactRepository; workspace-scoped URI handling in HTTP artifact operations.
CLI & Configuration
mlflow/cli/__init__.py, mlflow/environment_variables.py, mlflow/utils/workspace_utils.py, mlflow/utils/server_cli_utils.py, mlflow/utils/rest_utils.py
--enable-workspaces and --workspace-store-uri CLI flags; MLFLOW_ENABLE_WORKSPACES and MLFLOW_WORKSPACE_URI environment variables; workspace header injection in HTTP requests; resolve_entity_workspace_name() utility.
Client Integration
mlflow/tracking/client.py, mlflow/__init__.py, mlflow/tracking/_tracking_service/client.py, mlflow/tracking/default_experiment/registry.py, mlflow/tracking/fluent.py, mlflow/models/model.py
MlflowClient workspace methods (get_workspace_uri, list_workspaces, create/get/update/delete_workspace); fluent API exports; workspace-aware experiment resolution.
Documentation
docs/api_reference/api_inventory.txt, docs/api_reference/source/rest-api.rst, docs/docs/self-hosting/architecture/overview.mdx, docs/docs/self-hosting/index.mdx, docs/docs/self-hosting/workspaces/*, docs/sidebarsSelfHosting.ts
API inventory updates for workspace methods; REST API documentation; architecture overview with workspaces section; getting started, configuration, permissions, and workspace providers guides.
Workflow Tests
.github/workflows/master.yml
Added SQLAlchemy workspace store tests to CI matrix.
Core Tracking & Model Registry Tests
tests/store/tracking/test_sqlalchemy_store_workspace.py, tests/store/model_registry/test_sqlalchemy_store_workspace.py, tests/store/tracking/test_rest_store.py, tests/store/model_registry/test_rest_store.py
Comprehensive workspace isolation tests; cross-workspace access denial; artifact scoping; default workspace behavior; startup validation and migration scenarios.
Workspace Store Tests
tests/store/workspace/test_sqlalchemy_store.py, tests/store/workspace/test_rest_store.py, tests/store/workspace/test_workspace_validator.py
CRUD operations; error handling for not-found and conflicts; name validation; artifact root resolution.
Server & Auth Tests
tests/server/test_workspace_endpoints.py, tests/server/test_workspace_handlers.py, tests/server/auth/test_sqlalchemy_store.py, tests/server/auth/test_auth.py, tests/server/auth/test_client.py, tests/server/auth/db/test_cli.py
Workspace endpoint integration; permission management; default workspace experiment creation; workspace context lifecycle; RBAC enforcement.
Tracking Client & Utilities Tests
tests/tracking/test_client_workspace.py, tests/tracking/test_workspace_registry.py, tests/tracking/test_workspace_utils.py, tests/tracking/_model_registry/test_utils.py, tests/tracking/_tracking_service/test_utils.py, tests/utils/test_rest_utils.py, tests/utils/test_workspace_utils.py
URI resolution; provider registry; context management; REST header injection; entity workspace name resolution.
REST Tracking Integration Tests
tests/tracking/test_rest_tracking.py
Workspace-scoped REST/GraphQL endpoints; experiment/run/model isolation; OTLP with workspace context; permission-scoped access.
Entity Tests
tests/entities/test_experiment.py, tests/entities/test_evaluation_dataset.py, tests/entities/test_webhook.py, tests/entities/model_registry/test_model_version.py, tests/entities/model_registry/test_registered_model.py
Workspace property assertions; to_dict/from_dict round-trip validation; non-default workspace scenarios.
CLI & Server Tests
tests/test_cli.py, tests/server/test_init.py, tests/server/test_otel_api.py, tests/server/test_handlers.py, tests/tracking/test_mlflow_artifacts.py
Workspace CLI flags; artifacts-only and workspaces conflict; secret key handling; OTLP endpoint workspace context; RBAC resource resolution.
Test Helpers
tests/helpers/__init__.py, tests/helpers/db_mocks.py
Mock session maker for DB testing.
SQLAlchemy Store Tests
tests/store/tracking/test_sqlalchemy_store.py, tests/store/model_registry/test_sqlalchemy_store.py
Logged model parameter/tag handling on deleted models; error message updates for workspace scenarios.
GenAI Dataset Tests
tests/genai/datasets/test_fluent.py
Tag deletion error handling for missing tags.

Sequence Diagram(s)

sequenceDiagram
    participant Client as MLflow Client
    participant Tracking as Tracking Service
    participant WorkspaceCtx as Workspace Context
    participant Store as Tracking Store
    participant Auth as Auth Backend
    
    Client->>WorkspaceCtx: set_workspace("team-a")
    WorkspaceCtx->>WorkspaceCtx: Store in ContextVar
    
    Client->>Tracking: create_experiment("exp1")
    Tracking->>WorkspaceCtx: get_current_workspace()
    WorkspaceCtx-->>Tracking: "team-a"
    Tracking->>Store: create_experiment("exp1", workspace="team-a")
    Store->>Auth: check_permission(user, "team-a", "EDIT")
    Auth-->>Store: Permission granted
    Store->>Store: Insert experiment with workspace="team-a"
    Store-->>Tracking: Experiment created
    Tracking-->>Client: Experiment("exp1", workspace="team-a")
    
    Client->>WorkspaceCtx: set_workspace("team-b")
    Client->>Tracking: get_experiment(exp1_id)
    Tracking->>Store: get_experiment(exp1_id, workspace="team-b")
    Store->>Store: Query where workspace="team-b" AND id=exp1_id
    Store-->>Tracking: Not found (workspace isolation)
    Tracking-->>Client: MlflowException(RESOURCE_DOES_NOT_EXIST)
Loading
sequenceDiagram
    participant Server as MLflow Server
    participant Request as HTTP Request
    participant Handler as Workspace Handler
    participant AuthZ as AuthZ Service
    participant Handler2 as Request Handler
    
    Request->>Server: POST /api/2.0/mlflow/runs/search<br/>X-MLFLOW-WORKSPACE: team-a
    Server->>Handler: workspace_before_request_handler()
    Handler->>Handler: Extract workspace from header
    Handler->>Handler: Store in g.mlflow_workspace
    Handler->>AuthZ: Validate workspace access
    AuthZ-->>Handler: Authorized
    Handler-->>Server: Context set
    
    Server->>Handler2: Process request (search runs)
    Handler2->>Handler2: Filter runs by workspace="team-a"
    Handler2-->>Server: Return filtered runs
    
    Server->>Handler: workspace_teardown_request_handler()
    Handler->>Handler: Clear g.mlflow_workspace
    Handler-->>Server: Cleanup complete
Loading

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~150 minutes

Specific areas requiring extra attention:

  • Database migration logic (mlflow/server/auth/db/migrations/versions/2ed73881770d_workspace_permissions.py): Complex downgrade path with conflict detection; ensure rollback safety and data integrity
  • RBAC enforcement across endpoints (mlflow/server/auth/__init__.py, mlflow/server/workspace_helpers.py): Permission checks must correctly scope across workspaces; validate all code paths enforce isolation
  • Workspace context management (mlflow/tracking/_workspace/context.py, mlflow/store/workspace_aware_mixin.py): ContextVar usage must handle async/concurrent requests; verify token restoration logic
  • Artifact path scoping (mlflow/store/artifact/http_artifact_repo.py, mlflow/server/handlers.py): Workspace-scoped paths prevent cross-workspace access; verify URI construction prevents path traversal
  • Default workspace behavior (mlflow/store/workspace/sqlalchemy_store.py, mlflow/server/handlers.py): Transitions between workspace-enabled/disabled modes; ensure no orphaned data or constraint violations
  • REST API workspace header injection (mlflow/utils/rest_utils.py): Conditional header inclusion logic; verify correct endpoints receive/omit workspace header
  • Multi-workspace query filtering (mlflow/store/tracking/sqlalchemy_store.py, mlflow/store/model_registry/sqlalchemy_store.py): All queries must include workspace filters; search for incomplete scoping

Poem

🐰 Workspaces bloom like clover patches,
Teams now hidden behind safety hatches,
One server split, yet divided so neat,
Multi-tenant magic, oh what a treat!
Context and scopes keep the boundaries tight,
Permission gates guard each workspace right. 🌿

Pre-merge checks and finishing touches

❌ Failed checks (2 warnings, 1 inconclusive)
Check name Status Explanation Resolution
Description check ⚠️ Warning The pull request description is entirely missing; the author provided no content to describe the changes, objectives, testing approach, or documentation updates required. Add a comprehensive description following the repository template, including what changes are proposed, how the PR is tested, and whether documentation updates are needed.
Docstring Coverage ⚠️ Warning Docstring coverage is 41.18% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
Title check ❓ Inconclusive The title 'WIP: Test CI' is vague and non-descriptive, using generic terms that don't convey meaningful information about the changeset. Replace with a descriptive title that summarizes the main change, such as 'Add workspace multi-tenancy support' or 'Implement MLflow workspace functionality'.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@mprahl

mprahl commented Nov 21, 2025

Copy link
Copy Markdown
Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Nov 21, 2025

Copy link
Copy Markdown
✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@github-actions

Copy link
Copy Markdown

@mprahl Thank you for the contribution! Could you fix the following issue(s)?

⚠ Invalid PR template

This PR does not appear to have been filed using the MLflow PR template. Please copy the PR template from here and fill it out.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (4)
mlflow/server/handlers.py (1)

535-545: Workspace enablement incorrectly ignores MLFLOW_ARTIFACT_ROOT env var

The guard only checks default_artifact_root, but the error message promises that either --default-artifact-root or MLFLOW_ARTIFACT_ROOT is sufficient. If a user enables workspaces via --enable-workspaces and relies solely on MLFLOW_ARTIFACT_ROOT, this will incorrectly raise INVALID_PARAMETER_VALUE.

You can fix this by incorporating the env var into the check:

 def initialize_backend_stores(
     backend_store_uri: str | None = None,
     registry_store_uri: str | None = None,
     default_artifact_root: str | None = None,
     workspace_store_uri: str | None = None,
 ) -> None:
-    if MLFLOW_ENABLE_WORKSPACES.get() and not default_artifact_root:
-        raise MlflowException.invalid_parameter_value(
-            "--enable-workspaces requires --default-artifact-root or the "
-            "MLFLOW_ARTIFACT_ROOT environment variable to be set."
-        )
+    if MLFLOW_ENABLE_WORKSPACES.get():
+        from mlflow.server import ARTIFACT_ROOT_ENV_VAR
+        artifact_root = default_artifact_root or os.environ.get(ARTIFACT_ROOT_ENV_VAR)
+        if not artifact_root:
+            raise MlflowException.invalid_parameter_value(
+                "--enable-workspaces requires --default-artifact-root or the "
+                "MLFLOW_ARTIFACT_ROOT environment variable to be set."
+            )

This keeps behavior aligned with the documented contract while still enforcing a non‑null artifact root when workspaces are enabled.

mlflow/server/otel_api.py (1)

59-125: Fix Content-Type validation and align response payload with Content-Type header

The code has two confirmed issues:

  1. Content-Type check too strict (line 96): The exact match if content_type != "application/x-protobuf" rejects valid OTLP clients that send parameters (e.g., application/x-protobuf; charset=utf-8) or case variations. Following the pattern in mlflow/server/validation.py, split on ; and normalize case.

  2. Response Content-Type mismatch (line 103): The code sets response.headers["Content-Type"] = "application/x-protobuf" but returns a Pydantic OTelExportTraceServiceResponse() instance. FastAPI automatically serializes this to JSON, not protobuf. The actual response body will be JSON (e.g., {"partialSuccess": null}), not protobuf-encoded data.

Recommend either:

  • Parse Content-Type header correctly and return actual protobuf-encoded ExportTraceServiceResponse from opentelemetry.proto.collector.trace.v1.trace_service_pb2, OR
  • Change response Content-Type to application/json if returning JSON is intentional
mlflow/store/tracking/sqlalchemy_store.py (1)

648-710: _get_experiment default resolution per workspace is correct but may surprise legacy clients

The new logic:

  • Uses the active workspace when workspace is not provided.
  • For experiment_id is None and workspaces enabled, resolves (and lazily bootstraps) the default experiment by name within the current workspace, raising a clear error if creation fails.
  • For explicit experiment_id, always filters by both experiment_id and workspace.

This is the right behavior for multi-tenancy. As with the earlier comment, the workspace filter assumes the workspace column is populated for all experiments, so make sure migrations keep that invariant; otherwise old experiments with NULL workspace will be invisible here.

mlflow/server/auth/__init__.py (1)

820-835: filter_experiment_ids should use _has_experiment_read_access for workspace-aware filtering

This function currently bypasses workspace permission checks that are already implemented in _has_experiment_read_access. Other search filters like filter_search_experiments and filter_search_logged_models use this function to respect workspace isolation, but filter_experiment_ids only checks explicit experiment permissions and auth_config.default_permission.

When workspaces are enabled, this creates inconsistent filtering behavior: datasets and other resources filtered through filter_experiment_ids will not respect workspace-level access controls, while search operations will.

Refactor to call _has_experiment_read_access for each experiment ID (lines 810–835), passing the auth object and workspace permission getter, to ensure consistent workspace isolation.

🧹 Nitpick comments (53)
tests/server/test_workspace_handlers.py (5)

41-55: Consider consistent type hints in test doubles.

The DummyTrackingStore uses type hints for created_names and _experiments but not for method signatures. For better maintainability, consider adding type hints to all methods or removing them entirely from the test double.


82-108: Reduce duplication by extracting common test fixtures.

The DummyWorkspaceStore and DummyTrackingStore implementations are repeated across multiple tests with only minor variations. Consider extracting these into reusable fixtures with configurable behavior to improve maintainability.

For example, create a configurable fixture that accepts parameters for different behaviors:

@pytest.fixture
def dummy_workspace_store():
    class DummyWorkspaceStore:
        def __init__(self):
            self.deleted = False
            self.created = []
        
        def create_workspace(self, workspace):
            self.created.append(workspace.name)
            return workspace
        
        def delete_workspace(self, workspace_name):
            self.deleted = True
    
    return DummyWorkspaceStore()

Also applies to: 154-186, 208-239


172-174: Fragile filter string parsing in test doubles.

The substring check "name !=" in filter_string is brittle and implicit. Consider explicitly checking for the expected filter string value or documenting the assumption to make test expectations clearer.

For example:

-        if filter_string and "name !=" in filter_string:
+        # Filter out default experiment when searching for non-default experiments
+        if filter_string == f"name != '{Experiment.DEFAULT_EXPERIMENT_NAME}'":
             return []

Also applies to: 226-228


31-31: Consider extracting environment setup to a fixture.

The MLFLOW_ENABLE_WORKSPACES environment variable is set identically in all tests. Consider extracting this to a shared fixture to reduce duplication:

@pytest.fixture
def enable_workspaces(monkeypatch):
    monkeypatch.setenv(MLFLOW_ENABLE_WORKSPACES.name, "true")

Then use it in tests: def test_create_workspace_handler_creates_default_experiment(monkeypatch, enable_workspaces):

Also applies to: 80-80, 128-128, 152-152, 206-206, 256-256


145-148: Consider verifying workspace context cleanup after teardown.

The test calls workspace_teardown_request_handler but doesn't verify that the workspace context is properly cleared. Consider adding an assertion after teardown to ensure cleanup occurred:

         handlers.workspace_teardown_request_handler(None)
+        assert workspace_context.get_current_workspace() is None
mlflow/server/auth/config.py (1)

14-14: LGTM! Secure default will be applied.

The new boolean field is correctly typed and positioned. The secure default of False (applied in the reading logic below) follows the deny-by-default principle for access control.

Consider adding a docstring to the AuthConfig class or an inline comment explaining what this configuration option controls, for example:

 class AuthConfig(NamedTuple):
     default_permission: str
     database_uri: str
     admin_username: str
     admin_password: str
     authorization_function: str
+    # Controls whether users are automatically granted access to the default workspace
     grant_default_workspace_access: bool
tests/tracking/test_rest_tracking.py (3)

1672-1873: Extended non-local model version source coverage looks correct

The extended test_create_model_version_with_non_local_source:

  • Exercises a broad set of valid mlflow-artifacts: forms (various trailing slashes, host variants, multi-dot paths),
  • Asserts appropriate 400s for path-traversal-like and encoded traversal attempts across multiple schemes,
  • Adds store_type == "sqlalchemy" coverage for logged-model–based model_id sources vs disallowed file:// sources.

The structure and expectations are consistent with the intended security model for model version sources; no issues spotted. If duplication ever becomes a concern, you could factor the repeated requests.post + assertion patterns into small helpers, but that’s non-essential.


3429-3451: URL logging workspace behavior is well-covered (minor brittleness on private attribute)

The two new tests correctly distinguish:

  • Legacy /#/experiments/{exp_id} URLs when no workspace context is set, even if _supports_workspaces is True.
  • Workspace URLs including /#/workspaces/{workspace}/experiments/{exp_id} when a WorkspaceContext is active.

The only minor brittleness is relying on the private _supports_workspaces attribute; if that internal flag is renamed or removed, these tests will break even if external behavior is unchanged. That’s acceptable for internal tests but something to keep in mind.


4107-4591: REST workspace CRUD, scoping, and legacy endpoint tests are thorough; consider broadening caught exceptions

The new suite of REST workspace tests (CRUD, tracking, runs, artifacts, logged models, registry, default workspace, and legacy endpoints) provides very strong coverage of isolation semantics between workspaces and the default context. The patterns of:

  • Creating resources under multiple workspaces,
  • Asserting cross-workspace visibility failures (via exceptions or empty lists),
  • Verifying state in the original workspace after attempted cross-workspace operations,

are all sound and align with expected multi-tenant behavior.

One small robustness tweak:

  • In test_rest_run_operations_are_workspace_scoped, _call_safely only catches MlflowException, but the underlying client calls may raise RestException (or another exception type) on cross-workspace access, depending on how the client wraps errors. If you intend these calls to be treated as “expected failures” regardless of the precise exception subtype, it would be safer to broaden the catch:
-        def _call_safely(fn):
-            try:
-                fn()
-            except MlflowException:
-                pass
+        def _call_safely(fn):
+            try:
+                fn()
+            except (MlflowException, RestException):
+                # Cross-workspace access should fail; ignore the exact error shape
+                pass

This keeps the negative tests focused on state invariants without being sensitive to the exact exception class.

docs/api_reference/source/rest-api.rst (1)

62-67: Align error code formatting with rest of docs

The new guidance for workspaces under artifact_location looks good. For consistency with the rest of this page (e.g., RESOURCE_ALREADY_EXISTS), consider formatting the error code as an inline literal:

server uses the workspace's default artifact location and returns ``INVALID_PARAMETER_VALUE`` if provided.
mlflow/utils/rest_utils.py (1)

23-24: Workspace header resolution and injection are well-structured; consider minor hardening

The _resolve_active_workspace helper cleanly prefers the tracking workspace context and falls back to MLFLOW_WORKSPACE, trimming empty strings to avoid sending meaningless headers. _should_include_workspace_header’s path check to skip /mlflow/workspaces admin endpoints makes sense, and using headers.setdefault(WORKSPACE_HEADER_NAME, workspace) correctly preserves any explicitly provided workspace header (e.g., in extra_headers). One small hardening you might consider is explicitly asserting or coercing non‑string workspace values (should get_current_workspace() ever evolve to return something structured) to avoid accidentally sending non‑string objects as header values.

Also applies to: 44-45, 62-93, 157-160

tests/store/tracking/test_sqlalchemy_store.py (1)

7791-7807: Consider also asserting deleted model remains non‑retrievable after tagging

The test correctly verifies that tags written after delete_logged_model are persisted at the SqlLoggedModelTag level. For symmetry with test_log_logged_model_params_on_deleted_model, you might also assert that store.get_logged_model(model.model_id) still raises (i.e., tagging does not resurrect the model). This would tighten the contract without changing behavior.

For example:

with pytest.raises(MlflowException, match="not found"):
    store.get_logged_model(model.model_id)
mlflow/entities/evaluation_dataset.py (1)

326-347: Harden to_df error-code matching for dataset-not-found cases

to_df currently interprets a missing dataset by checking:

error_code = getattr(e, "error_code", None)
if error_code == ErrorCode.Name(RESOURCE_DOES_NOT_EXIST):
    ...

This assumes e.error_code is always the string name "RESOURCE_DOES_NOT_EXIST". If any store surfaces the numeric enum value instead, this branch would be skipped and the exception re-raised.

To tolerate both representations, consider:

-        except MlflowException as e:
-            error_code = getattr(e, "error_code", None)
-            if error_code == ErrorCode.Name(RESOURCE_DOES_NOT_EXIST):
+        except MlflowException as e:
+            error_code = getattr(e, "error_code", None)
+            not_found_codes = {
+                RESOURCE_DOES_NOT_EXIST,
+                ErrorCode.Name(RESOURCE_DOES_NOT_EXIST),
+            }
+            if error_code in not_found_codes:
                 # Cache the empty records list to avoid repeated store lookups
                 self._records = []
                 records = []
             else:
                 raise

This keeps behavior unchanged where the string is used, but also handles numeric codes if any backend returns them.

mlflow/store/workspace/abstract_store.py (1)

64-76: Abstract workspace store and name validation are well-structured (minor DRY opportunity)

The AbstractStore interface plus resolve_artifact_root default (default_artifact_root, True) give providers a clear extension point without forcing per-workspace customization. WorkspaceNameValidator correctly enforces length, pattern, and reserved-name checks with good error messages.

If you want to reduce maintenance overhead, you could have is_valid() delegate to validate() and catch MlflowException, so that future rule changes only need to be updated in one place. Not critical, but it avoids the two methods drifting apart.

Also applies to: 79-128

mlflow/server/handlers.py (1)

2935-2963: Workspace‑scoped artifact paths: behavior is sound

The _workspace_scoped_repo_path helper and its use in download/upload/list/delete and multipart endpoints correctly:

  • Leave paths untouched when no active workspace.
  • Prefix with workspaces/<workspace> when a workspace is active, isolating proxied artifacts per tenant.
  • Avoid double‑prefixing when callers already pass workspaces/….

One edge case to consider:

  • For artifact_path == "workspaces" (without a trailing /), normalized.startswith("workspaces/") is false, so you'll end up with workspaces/<ws>/workspaces. If that path is ever used externally, normalize this case:
-    if normalized.startswith("workspaces/"):
+    if normalized == "workspaces" or normalized.startswith("workspaces/"):
         return normalized or posixpath.join("workspaces", workspace)

The multipart upload methods already accept the artifact_path parameter, so passing repo_path uses the existing interface correctly—no signature extension needed.

Also applies to: 2984-2999, 3010-3016, 3033-3038

tests/tracking/test_workspace_registry.py (1)

22-27: Consider using a public cleanup method.

Line 27 accesses the private _engine attribute for cleanup. While this works for testing, it couples the test to implementation details. Consider adding a public dispose() or close() method to SqlAlchemyStore if one doesn't already exist.

tests/tracking/test_workspace_utils.py (1)

1-38: Workspace URI resolution tests cover key precedence rules

The autouse fixture correctly cleans both in-memory and env state, and the tests clearly exercise explicit, configured, env, and tracking-default resolution paths. The redundancy of resetting state both before and after is harmless and keeps isolation explicit.

tests/entities/model_registry/test_model_version.py (1)

7-44: ModelVersion workspace tests are thorough; avoid hardcoding the default name

The extended _check helper and new round-trip test give good coverage of workspace across construction, dict/proto conversions, and __str__. In test_string_repr, consider using DEFAULT_WORKSPACE_NAME instead of the literal 'default' so the test stays aligned if the default workspace name ever changes:

-        "status='PENDING_REGISTRATION', status_message='Copying!', tags={}, user_id='user one', "
-        "version='43', workspace='default'>"
+        "status='PENDING_REGISTRATION', status_message='Copying!', tags={}, user_id='user one', "
+        f"version='43', workspace='{DEFAULT_WORKSPACE_NAME}'>"

Also applies to: 88-108, 159-185, 188-214

mlflow/tracking/default_experiment/registry.py (1)

63-71: Return type annotation may limit supported Python versions

get_experiment_id is annotated as -> str | None, which relies on PEP 604 union syntax and only parses on Python 3.10+. If this branch still targets Python 3.8/3.9, this will be a syntax error. In that case, prefer the older spelling:

-from typing import Optional
-
-def get_experiment_id() -> str | None:
+from typing import Optional
+
+def get_experiment_id() -> Optional[str]:

If you have already dropped support for <3.10, this is fine as-is, but it’s worth confirming against your supported Python matrix.

tests/helpers/db_mocks.py (1)

7-22: Session maker mock matches expected pattern; optional room for richer query mocks

This helper cleanly mimics a managed session maker and produces chainable query.filter().order_by().first() calls that return None, which should be sufficient for the current tests.

If future tests need additional query behaviors (e.g., filter_by, all, or configurable first() values), consider extending _mock_query accordingly or parameterizing the mock, but it’s not necessary for this PR.

tests/entities/model_registry/test_registered_model.py (1)

5-6: Workspace coverage for RegisteredModel is solid; consider reusing the default constant in repr assertion

The additions to _check, the as_dict fixtures, and the new test_registered_model_non_default_workspace_round_trip give good coverage that the workspace field is preserved across construction, dict(rmd), from_dictionary, and __str__.

One minor polish point: in test_string_repr, the expected string currently hard-codes workspace='default'. To avoid drifting from DEFAULT_WORKSPACE_NAME if that constant ever changes, you could build the expected string with the constant, e.g. using an f-string:

-from mlflow.utils.workspace_utils import DEFAULT_WORKSPACE_NAME
+from mlflow.utils.workspace_utils import DEFAULT_WORKSPACE_NAME
@@
-    assert (
-        str(rmd) == "<RegisteredModel: aliases={}, creation_timestamp=1000, "
-        "deployment_job_id=None, deployment_job_state=None, description='something about a model',"
-        " last_updated_timestamp=2002, "
-        "latest_versions=['1', '2', '3'], name='myname', tags={}, workspace='default'>"
-    )
+    expected = (
+        "<RegisteredModel: aliases={}, creation_timestamp=1000, "
+        "deployment_job_id=None, deployment_job_state=None, "
+        "description='something about a model', last_updated_timestamp=2002, "
+        "latest_versions=['1', '2', '3'], name='myname', tags={}, "
+        f"workspace='{DEFAULT_WORKSPACE_NAME}'>"
+    )
+    assert str(rmd) == expected

Not required for correctness, but it keeps tests aligned with the shared constant.

Also applies to: 19-20, 30-31, 39-50, 93-104, 125-136, 152-163, 174-189, 192-212

docs/docs/self-hosting/workspaces/index.mdx (1)

1-152: Workspace docs are clear; please confirm they exactly match the current implementation

The page gives a good, structured explanation of workspaces, isolation, artifact layout, and migration behavior, and it aligns conceptually with the code changes (e.g., default workspace name, workspaces/<workspace-name> artifact prefix).

Given this is a WIP, I’d recommend a quick pass to verify that:

  • The enablement mechanisms and names (MLFLOW_ENABLE_WORKSPACES, --enable-workspaces, MLFLOW_WORKSPACE) match the actual flags / environment variables in the server code.
  • The “Workspace Resolution” order matches the combined behavior of mlflow.set_workspace, MLFLOW_WORKSPACE, and any get_default_workspace() usage (e.g., via resolve_entity_workspace_name and get_default_workspace_optional).

This is just to avoid subtle drift between docs and behavior.

mlflow/store/artifact/http_artifact_repo.py (2)

39-59: Workspace parsing hook is consistent but currently unused; consider tightening the API

The new _get_workspace_scoped_artifact_uri correctly detects a /mlflow-artifacts/artifacts/workspaces/<workspace>/… segment without changing existing behavior, and _host_creds still uses the original artifact_uri, so there’s no regression.

Right now, though, the parsed workspace_component isn’t used and _get_artifacts_endpoint_prefix(_workspace_component) ignores its argument. To keep things simpler until you actually need workspace-specific endpoint prefixes, you might either:

  • Drop the parameter on _get_artifacts_endpoint_prefix for now, or
  • Thread workspace_component through and implement a distinct prefix when/if the server API requires it.

Not a blocker, but it will make the intent clearer to future readers.


101-105: List and multipart endpoints preserve workspace-scoped paths; small refactor opportunity

The updated list_artifacts and _construct_mpu_uri_and_path both derive the base URL and “root” path by splitting on the artifacts endpoint prefix. That continues to work for both:

  • Legacy URIs: /mlflow-artifacts/artifacts/<root>
  • Workspace URIs: /mlflow-artifacts/artifacts/workspaces/<workspace>/<root>

since the tail (and thus root) simply includes the workspaces/<workspace>/… path segment.

If you want to reduce duplication and future-proof this, consider centralizing the split logic into a helper that returns (base_url, root_path, workspace_component) and reuse it in both list_artifacts and _construct_mpu_uri_and_path. This would also make it easier to change the underlying endpoint layout later in one place.

Also applies to: 136-140

tests/store/model_registry/test_rest_store.py (1)

161-175: Workspace not-supported error path test looks solid but is tightly coupled to internals

The test correctly exercises the failure path when _workspace_support is False and a non-empty workspace is resolved, including checking the exception message. It does, however, depend on private details (store._workspace_support and base_rest_store.rest_utils._resolve_active_workspace), so it may become brittle if internal wiring changes. If the underlying implementation ever exposes a public way to simulate “no workspace support”, consider switching to that to reduce coupling.

docs/docs/self-hosting/workspaces/workspace-providers.mdx (1)

1-137: Docs clearly describe workspace providers; consider aligning snippets with actual APIs

The overall explanation of provider architecture, discovery, default SQL provider behavior, artifact for_workspace() hooks, and Flask context usage is clear and useful. To avoid confusion for users, it’s worth double-checking that:

  • The AbstractStore and ArtifactRepository.for_workspace() signatures in the snippets match the actual implementations.
  • The Workspace import path in the example (from mlflow.entities import Workspace) aligns with the public API you expect users to use (tests here import Workspace from mlflow.entities.workspace).

If they differ, updating the examples to mirror the real interfaces will keep the docs from drifting.

tests/store/tracking/test_rest_store.py (1)

2862-2880: log_spans version checking and caching tests are comprehensive but one subtest is brittle

Pros:

  • test_log_spans_with_version_check exercises:
    • failure to retrieve version,
    • version < 3.4 rejecting,
    • 3.4 / 3.5 success paths.
  • test_server_version_check_caching confirms _get_server_version is invoked once per host and that later log_spans calls reuse the cached version (even across RestStore instances).

Concern:

  • The “real timeout test” (Test 5 within test_log_spans_with_version_check) relies on an actual connection attempt to https://host5 and asserts elapsed time < 5s. This can become flaky under slow DNS or unusual network conditions.

Consider stubbing rest_utils.http_request with a time.sleep to simulate timeout instead of performing a real network call; behavior would be deterministic while still validating timeout semantics.

Also applies to: 2882-2947, 2949-3021

mlflow/store/workspace/rest_store.py (1)

16-18: REST workspace store CRUD looks correct; consider calling super().__init__()

  • Endpoints and methods (GET/POST/PATCH/DELETE /api/2.0/mlflow/workspaces[...]) and the use of verify_rest_response are consistent and correct.
  • create_workspace explicitly maps 400/409 + RESOURCE_ALREADY_EXISTS into MlflowException(..., RESOURCE_ALREADY_EXISTS), which matches MLflow’s error semantics.
  • Handling of empty response bodies by falling back to the request values is pragmatic and avoids extra server requirements.
  • get_default_workspace explicitly raising an invalid-parameter exception is clear about this provider’s capabilities.

One small robustness improvement:

 class RestWorkspaceStore(AbstractStore):
     """REST-backed workspace store implementation."""

     def __init__(self, get_host_creds):
-        self.get_host_creds = get_host_creds
+        super().__init__()
+        self.get_host_creds = get_host_creds

Calling super().__init__() future-proofs the class if AbstractStore later adds initialization logic.

Also applies to: 19-106

mlflow/server/auth/db/models.py (1)

104-125: New SqlWorkspacePermission model is straightforward; consider optional FK to users

  • Composite primary key on (workspace, username, resource_type) plus indices on username and workspace should support common lookup patterns efficiently.
  • to_mlflow_entity() correctly populates WorkspacePermission.

Depending on how strictly you want to enforce referential integrity between workspace_permissions.username and users.username, you might optionally add a ForeignKey("users.username") and/or a relationship on SqlUser. If usernames can be non-user principals, the current design is fine as is.

mlflow/server/otel_api.py (1)

1-11: Update module docstring to reflect non-placeholder implementation

The docstring still describes this as a minimal placeholder and implies span ingestion is not fully implemented, which is no longer accurate given the concrete ingestion and conversion logic below. Consider updating it to describe the actual behavior and workspace-aware handling to avoid confusion.

mlflow/cli/__init__.py (1)

587-597: Avoid duplicating workspace env setup via both os.environ and EnvironmentVariable.set

You’re setting workspace-related env in two ways:

  • Directly via os.environ[...] = ... above.
  • Via MLFLOW_ENABLE_WORKSPACES.set(enable_workspaces) and MLFLOW_WORKSPACE_URI.set(...) here.

Since the EnvironmentVariable helpers already write to os.environ, maintaining both paths adds redundancy and a chance for drift if behavior changes in one place. Consider consolidating on the EnvironmentVariable helpers for all writes and keeping the logging/echo behavior here.

mlflow/server/auth/entities.py (2)

148-151: RegisteredModelPermission workspace handling looks sound; consider explicit type-level docs

The added workspace parameter defaults to None, is normalized via resolve_entity_workspace_name, and is round-tripped through to_json / from_json with a backward-compatible dictionary.get("workspace"). That keeps existing JSON payloads and positional call sites working while making workspace explicit. It might be worth documenting (in a docstring or higher-level docs) that omitting workspace in JSON will resolve to the default workspace so callers understand the behavior, but the implementation itself is solid.

Also applies to: 155-157, 175-181, 184-190


244-288: Align WorkspacePermission workspace normalization and validation with other entities

WorkspacePermission currently stores workspace as passed and validates required fields using if not all([...]). For consistency and to avoid surprising falsy-value behavior:

  • Consider normalizing workspace via resolve_entity_workspace_name (and possibly reusing WorkspaceNameValidator) so this entity behaves like RegisteredModelPermission.
  • Prefer explicit is None checks over all([...]) so that values like "0" or numeric permission codes don't get rejected purely because they may be falsy in future.

This keeps workspace semantics and validation consistent across auth entities.

mlflow/store/workspace/sqlalchemy_store.py (1)

49-65: create_workspace behavior and error mapping look correct; consider trimming DB error detail

create_workspace validates the name, persists via a managed session, and maps IntegrityError to MlflowException with RESOURCE_ALREADY_EXISTS, which matches the tests and MLflow patterns. The message currently appends the raw DB exception (Error: {exc}); if you care about not exposing backend details to API consumers, you might drop that suffix and rely on logs instead, but functionally this is fine.

tests/server/test_workspace_endpoints.py (1)

59-72: Strengthen endpoint tests by asserting store call arguments

The endpoint tests currently verify status codes and response payloads and only assert that create_workspace, update_workspace, and delete_workspace were called once. To better lock in the handler contract, consider asserting the arguments as well, for example:

  • For create: inspect mock_workspace_store.create_workspace.call_args and assert the Workspace passed has the expected name and description.
  • For update: similarly validate the Workspace argument (or separate name/description parameters if you change the store signature).
  • For delete: you already assert "team-e" is passed, which is good; mirroring this pattern for create/update would improve coverage.

This would catch regressions where request JSON isn’t correctly mapped into the workspace entity.

Also applies to: 85-97, 100-105

mlflow/tracking/_workspace/fluent.py (1)

36-50: set_workspace behavior is reasonable; consider marking it experimental like other public APIs

The set_workspace implementation is thread-safe, correctly handles None by clearing both in-process context and MLFLOW_WORKSPACE, and validates non-default names with WorkspaceNameValidator. The CRUD helpers are consistently funneled through _workspace_client_call and apply name validation except for the reserved default, which aligns with store semantics.

One minor consistency point: set_workspace is exported via __all__ but is not annotated with @experimental(version="3.7.0") like the other public functions in this module. If the intention is for all newly introduced workspace APIs to be experimental, you may want to add the decorator to set_workspace as well; otherwise, it effectively appears more stable than the others.

Also applies to: 52-94, 96-104

tests/store/model_registry/test_sqlalchemy_store_workspace.py (1)

36-105: Consider asserting error codes for all cross‑workspace failure paths

For many cross‑workspace operations you assert both the error message and error_code == "RESOURCE_DOES_NOT_EXIST", but a few (e.g., rename_registered_model / delete_registered_model in the "team-b" context, and some metadata operations) only assert on the message. For long‑term stability, you may want to assert the error code in those places as well so behavior remains consistent even if messages change.

Also applies to: 141-173, 228-252

mlflow/tracking/client.py (1)

1872-1885: Optional: enforce artifact_location restriction when workspaces are enabled

The create_experiment docstring now states that artifact_location cannot be specified when workspaces are enabled because a workspace‑scoped default is always used, but the method still passes the argument through unconditionally to the tracking store. If you want failures to be more immediate and client‑side, consider adding a guard like:

     def create_experiment(
         self,
         name: str,
         artifact_location: str | None = None,
         tags: dict[str, Any] | None = None,
     ) -> str:
@@
-        """
-        Create an experiment.
+        """
+        Create an experiment.
@@
-        """
-        return self._tracking_client.create_experiment(name, artifact_location, tags)
+        """
+        # When workspaces are enabled, artifact locations are determined by the workspace.
+        if workspace_utils.workspaces_enabled() and artifact_location is not None:
+            raise MlflowException(
+                "artifact_location cannot be specified when workspaces are enabled; "
+                "a workspace-scoped default artifact location is always used.",
+                error_code=INVALID_PARAMETER_VALUE,
+            )
+        return self._tracking_client.create_experiment(name, artifact_location, tags)

(Adjust helper / error code usage to match workspace_utils and existing validation utilities.) If you prefer to keep this server‑side only, the current implementation is still consistent with the updated documentation.

Also applies to: 1924-1925

mlflow/tracking/_workspace/registry.py (1)

37-47: Optional: consider broadening entrypoint error handling or logging detail

register_entrypoints() currently catches only AttributeError and ImportError when loading workspace provider plugins, emitting a warning. If a provider raises another exception type during load (e.g., misconfiguration), initialization will fail hard. Depending on how defensive you want plugin loading to be, you might consider catching Exception here and including provider name and exception type in the warning to avoid taking down the process due to a single bad plugin.

mlflow/store/tracking/sqlalchemy_store.py (8)

449-527: Artifact location derivation and isolation checks are well thought out

_get_artifact_location now parameterizes on workspace and correctly:

  • Uses the provider’s resolved root when workspaces are enabled, with an optional /workspaces/<name> prefix when should_append is True.
  • Falls back to artifact_root_uri when the provider doesn’t implement resolution or returns None.

_validate_artifact_isolation_constraints and the reserved prefix checks around /workspaces are a sensible guardrail to prevent enabling workspaces on top of conflicting pre-existing artifact layouts. The startup scan also skips heavy checks when non-default workspaces already exist, which is a nice optimization for already-migrated installs.


528-548: Single-tenant guard _ensure_no_non_default_workspace_experiments is correct but message could mention workspace flag

The method properly fails fast if any experiment has workspace != DEFAULT_WORKSPACE_NAME, preventing silently running in single-tenant mode on a DB that already has multi-workspace data. You might consider explicitly mentioning the MLFLOW_ENABLE_WORKSPACES (or equivalent) flag in the error message to make remediation clearer to operators, but the current wording is already actionable.


891-921: _get_run_inputs workspace scoping is sound but always uses the active workspace

Joining through SqlExperiment and filtering SqlExperiment.workspace == workspace in _get_run_inputs ensures dataset inputs on runs in other workspaces are not exposed. Note that this uses self._get_active_workspace() rather than the workspace bound to any outer _workspace_session. That’s okay today (workspace context is global), but if you ever introduce cross-workspace administrative views, you might want to thread an explicit workspace argument instead of re-reading global context.


1424-1458: Bulk metric history properly filters run_ids by workspace when enabled

In get_metric_history_bulk, the allowed_run_ids derivation via _runs_query under _workspaces_enabled() ensures that only runs in the active workspace contribute metrics. The early return when allowed_run_ids is empty is correct and avoids unnecessary work. This does add an extra query in workspace mode, but it’s acceptable given the isolation requirements.


1976-2109: _log_inputs_impl enforces dataset/run workspace consistency but has a minor key inconsistency

Using workspace = self._get_active_workspace() up front and then:

  • Joining SqlDatasetSqlExperiment and filtering SqlExperiment.workspace == workspace to detect existing datasets.
  • Storing new SqlDataset rows with the calling experiment_id.
  • Creating SqlInput edges without workspace columns (since dataset_id and run_id are already scoped)

is a solid pattern for multi-tenancy.

Minor nit: input_uuids is keyed as (source_id, destination_id) for existing inputs but uses (dataset_input.dataset.name, dataset_input.dataset.digest) when populating new entries. Since membership checks later use (dataset_uuid, run_id), the keys for new entries are never read and are effectively dead bookkeeping. It doesn’t break correctness (datasets are already deduped by (name,digest)), but aligning the key shape would improve clarity.

-            for existing_input in existing_inputs:
-                input_uuids[(existing_input.source_id, existing_input.destination_id)] = (
-                    existing_input.input_uuid
-                )
+            for existing_input in existing_inputs:
+                input_uuids[(existing_input.source_id, existing_input.destination_id)] = (
+                    existing_input.input_uuid
+                )
 ...
-                if (dataset_uuid, run_id) not in input_uuids:
+                if (dataset_uuid, run_id) not in input_uuids:
                     new_input_uuid = uuid.uuid4().hex
-                    input_uuids[(dataset_input.dataset.name, dataset_input.dataset.digest)] = (
-                        new_input_uuid
-                    )
+                    input_uuids[(dataset_uuid, run_id)] = new_input_uuid

2127-2208: Model input/output helpers correctly adopt workspace-aware run filtering

_get_model_inputs and _get_model_outputs_bulk both:

  • Reuse _runs_query under _workspaces_enabled() when deriving allowed run_ids.
  • Fall back to identity mappings for missing run_ids to preserve API shape.

The recursion in _get_model_inputs when session is None is simple and safe, but if you ever extend the method, consider splitting the internal implementation into a non-recursive private helper (e.g. _get_model_inputs_in_session) for clarity.


4575-4657: upsert_dataset_records correctly reuses dataset.workspace in the final update

After using _get_dataset_record to fetch dataset within the correct workspace, the final update on SqlEvaluationDataset includes both dataset_id and dataset.workspace in the filter. While dataset_id is already globally unique (UUID-derived), this extra predicate is harmless and future-proofs against any accidental reuse patterns. The schema/profile recomputation is entirely local to this dataset.


4671-4687: get_dataset_experiment_ids is workspace-safe but could optionally filter experiments

When workspaces are enabled, the upfront _get_dataset_record call inside a managed session is a good guard; returning [] when it fails avoids leaking associations for datasets that belong to other workspaces. The subsequent search_entities_by_source call is global, but because dataset_id is globally unique and associations are only created for experiments validated in the same workspace, this is effectively safe. If you ever relax those invariants, consider adding a join to SqlExperiment with a workspace filter here as well.

mlflow/server/auth/sqlalchemy_store.py (2)

35-47: _WORKSPACE_RESOURCE_TYPES and validation guard against unsupported resource types

Defining _WORKSPACE_RESOURCE_TYPES = {"*", "experiments", "registered_models"} and enforcing it in _validate_workspace_resource_type() ensures you don’t accidentally persist or reason about undefined resource types. The error message enumerating valid types should be very helpful at API boundaries.


389-407: list_accessible_workspace_names correctly aggregates readable workspaces via wildcard support

Given a username:

  • It pulls all SqlWorkspacePermission rows for [username, "*"].
  • Validates each permission and uses get_permission(permission).can_read to decide whether the workspace is accessible.
  • Returns a deduplicated set[str] of workspace names.

This honors wildcard-user permissions and any resource type (since any readable resource implies the workspace is “accessible”), which is a reasonable definition for workspace listing.

mlflow/server/auth/__init__.py (4)

185-195: AuthContext helper is simple and safe; consider documenting expected attributes

Using AuthContext as a thin wrapper over Authorization is fine and makes later code cleaner. Since you depend on auth.username and sometimes auth.is_admin, a short docstring on AuthContext (and/or on the expected shape of Authorization) would help future custom auth implementations know which attributes they must provide.

Please double-check any custom authorization_function you may have configured still returns an object exposing a username attribute, as previously assumed.


505-613: Experiment / run / model / scorer permission getters correctly pass workspace fallbacks

The updated _get_permission_from_* helpers now:

  • Authenticate once per call and derive AuthContext.
  • Call _get_permission_from_store_or_default with a workspace_fallback pointing at the appropriate _workspace_permission_for_* helper.
    That wiring looks consistent across experiments, runs, logged models, registered models, and scorers.

To avoid repeated authenticate_request() calls and redundant AuthContext construction across these helpers, consider a small internal helper that takes (experiment_id | model_name, resource_type) and returns (auth, auth_ctx) plus a bound workspace_fallback.

Please re-run the auth regression tests for all affected endpoints (experiment / run / logged model / registered model / scorer) to validate there are no behavioral differences when MLFLOW_ENABLE_WORKSPACES is unset.


711-748: Creation of experiments and registered models is now workspace-scoped

validate_can_create_experiment / validate_can_create_registered_model:

  • Preserve the historical “always allowed” behavior when workspaces are disabled.
  • In workspace mode, require either admin or MANAGE permission in the current workspace.
  • Return False when no active workspace is set.
    This is a good tightening of semantics for multi-tenant deployments.

You might want to log a debug-level message when workspace_name is None in workspace mode, to make misconfigured clients (not setting a workspace) easier to diagnose.

Please ensure client-side workspace selection helpers (e.g., mlflow.set_workspace / MlflowClient.set_workspace) are documented as required before calling creation APIs in workspace mode.


1410-1465: filter_list_workspaces safely handles malformed responses and enforces per-user visibility

The response filter:

  • Short-circuits for admins.
  • Gracefully handles JSON decoding failures and malformed workspaces payloads by returning an empty list instead of erroring.
  • Uses list_accessible_workspace_names plus grant_default_workspace_access and default workspace resolution to compute allowed names.
  • Never exposes workspaces if the auth store claims to support workspaces but lacks permission data (logs a warning and returns none).
    This is a good defense-in-depth design.

You might consider logging the total number of filtered-out workspaces at debug level to aid troubleshooting in multi-tenant environments.

Please verify that list_accessible_workspace_names is efficient enough to be called on every list-workspaces response, especially in environments with many workspaces.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 5b68933 and d6b41b1.

📒 Files selected for processing (103)
  • .github/workflows/master.yml (1 hunks)
  • docs/api_reference/api_inventory.txt (14 hunks)
  • docs/api_reference/source/rest-api.rst (1 hunks)
  • docs/docs/self-hosting/architecture/overview.mdx (1 hunks)
  • docs/docs/self-hosting/index.mdx (1 hunks)
  • docs/docs/self-hosting/workspaces/configuration.mdx (1 hunks)
  • docs/docs/self-hosting/workspaces/getting-started.mdx (1 hunks)
  • docs/docs/self-hosting/workspaces/index.mdx (1 hunks)
  • docs/docs/self-hosting/workspaces/permissions.mdx (1 hunks)
  • docs/docs/self-hosting/workspaces/workspace-providers.mdx (1 hunks)
  • docs/sidebarsSelfHosting.ts (1 hunks)
  • mlflow/__init__.py (5 hunks)
  • mlflow/cli/__init__.py (6 hunks)
  • mlflow/entities/evaluation_dataset.py (7 hunks)
  • mlflow/entities/experiment.py (6 hunks)
  • mlflow/entities/model_registry/model_version.py (4 hunks)
  • mlflow/entities/model_registry/registered_model.py (6 hunks)
  • mlflow/entities/webhook.py (7 hunks)
  • mlflow/entities/workspace.py (1 hunks)
  • mlflow/environment_variables.py (3 hunks)
  • mlflow/models/model.py (2 hunks)
  • mlflow/server/__init__.py (4 hunks)
  • mlflow/server/auth/__init__.py (35 hunks)
  • mlflow/server/auth/basic_auth.ini (1 hunks)
  • mlflow/server/auth/client.py (4 hunks)
  • mlflow/server/auth/config.py (2 hunks)
  • mlflow/server/auth/db/migrations/versions/2ed73881770d_workspace_permissions.py (1 hunks)
  • mlflow/server/auth/db/models.py (4 hunks)
  • mlflow/server/auth/db/utils.py (0 hunks)
  • mlflow/server/auth/entities.py (5 hunks)
  • mlflow/server/auth/routes.py (1 hunks)
  • mlflow/server/auth/sqlalchemy_store.py (6 hunks)
  • mlflow/server/handlers.py (21 hunks)
  • mlflow/server/otel_api.py (6 hunks)
  • mlflow/server/workspace_helpers.py (1 hunks)
  • mlflow/store/artifact/artifact_repo.py (2 hunks)
  • mlflow/store/artifact/artifact_repository_registry.py (2 hunks)
  • mlflow/store/artifact/http_artifact_repo.py (4 hunks)
  • mlflow/store/db/utils.py (2 hunks)
  • mlflow/store/model_registry/abstract_store.py (1 hunks)
  • mlflow/store/model_registry/base_rest_store.py (3 hunks)
  • mlflow/store/model_registry/dbmodels/models.py (7 hunks)
  • mlflow/store/model_registry/sqlalchemy_store.py (37 hunks)
  • mlflow/store/tracking/abstract_store.py (1 hunks)
  • mlflow/store/tracking/dbmodels/models.py (6 hunks)
  • mlflow/store/tracking/rest_store.py (5 hunks)
  • mlflow/store/tracking/sqlalchemy_store.py (89 hunks)
  • mlflow/store/workspace/__init__.py (1 hunks)
  • mlflow/store/workspace/abstract_store.py (1 hunks)
  • mlflow/store/workspace/dbmodels/models.py (1 hunks)
  • mlflow/store/workspace/rest_store.py (1 hunks)
  • mlflow/store/workspace/sqlalchemy_store.py (1 hunks)
  • mlflow/store/workspace/utils.py (1 hunks)
  • mlflow/store/workspace_aware_mixin.py (1 hunks)
  • mlflow/tracking/_tracking_service/client.py (6 hunks)
  • mlflow/tracking/_workspace/__init__.py (1 hunks)
  • mlflow/tracking/_workspace/client.py (1 hunks)
  • mlflow/tracking/_workspace/context.py (1 hunks)
  • mlflow/tracking/_workspace/fluent.py (1 hunks)
  • mlflow/tracking/_workspace/registry.py (1 hunks)
  • mlflow/tracking/_workspace/utils.py (1 hunks)
  • mlflow/tracking/client.py (6 hunks)
  • mlflow/tracking/default_experiment/registry.py (2 hunks)
  • mlflow/tracking/fluent.py (2 hunks)
  • mlflow/utils/rest_utils.py (6 hunks)
  • mlflow/utils/server_cli_utils.py (1 hunks)
  • mlflow/utils/workspace_utils.py (1 hunks)
  • tests/entities/model_registry/test_model_version.py (5 hunks)
  • tests/entities/model_registry/test_registered_model.py (8 hunks)
  • tests/entities/test_evaluation_dataset.py (1 hunks)
  • tests/entities/test_experiment.py (4 hunks)
  • tests/entities/test_webhook.py (2 hunks)
  • tests/genai/datasets/test_fluent.py (1 hunks)
  • tests/helpers/__init__.py (1 hunks)
  • tests/helpers/db_mocks.py (1 hunks)
  • tests/server/auth/db/test_cli.py (3 hunks)
  • tests/server/auth/test_auth.py (6 hunks)
  • tests/server/auth/test_client.py (5 hunks)
  • tests/server/auth/test_sqlalchemy_store.py (9 hunks)
  • tests/server/test_handlers.py (3 hunks)
  • tests/server/test_init.py (3 hunks)
  • tests/server/test_otel_api.py (1 hunks)
  • tests/server/test_workspace_endpoints.py (1 hunks)
  • tests/server/test_workspace_handlers.py (1 hunks)
  • tests/store/model_registry/test_rest_store.py (1 hunks)
  • tests/store/model_registry/test_sqlalchemy_store.py (2 hunks)
  • tests/store/model_registry/test_sqlalchemy_store_workspace.py (1 hunks)
  • tests/store/tracking/test_rest_store.py (12 hunks)
  • tests/store/tracking/test_sqlalchemy_store.py (3 hunks)
  • tests/store/tracking/test_sqlalchemy_store_workspace.py (1 hunks)
  • tests/store/workspace/test_rest_store.py (1 hunks)
  • tests/store/workspace/test_sqlalchemy_store.py (1 hunks)
  • tests/store/workspace/test_workspace_validator.py (1 hunks)
  • tests/test_cli.py (29 hunks)
  • tests/tracking/_model_registry/test_utils.py (2 hunks)
  • tests/tracking/_tracking_service/test_utils.py (3 hunks)
  • tests/tracking/test_client_workspace.py (1 hunks)
  • tests/tracking/test_mlflow_artifacts.py (1 hunks)
  • tests/tracking/test_rest_tracking.py (13 hunks)
  • tests/tracking/test_workspace_registry.py (1 hunks)
  • tests/tracking/test_workspace_utils.py (1 hunks)
  • tests/utils/test_rest_utils.py (3 hunks)
  • tests/utils/test_workspace_utils.py (1 hunks)
💤 Files with no reviewable changes (1)
  • mlflow/server/auth/db/utils.py
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (27)
  • GitHub Check: r
  • GitHub Check: build
  • GitHub Check: pyfunc (4)
  • GitHub Check: pyfunc (3)
  • GitHub Check: protect
  • GitHub Check: java
  • GitHub Check: windows (3)
  • GitHub Check: windows (1)
  • GitHub Check: windows (4)
  • GitHub Check: windows (2)
  • GitHub Check: pyfunc (2)
  • GitHub Check: pyfunc (1)
  • GitHub Check: flavors
  • GitHub Check: evaluate (1)
  • GitHub Check: evaluate (2)
  • GitHub Check: typescript-sdk (22)
  • GitHub Check: genai
  • GitHub Check: models (1)
  • GitHub Check: models (2)
  • GitHub Check: database
  • GitHub Check: build (tracing)
  • GitHub Check: python-skinny
  • GitHub Check: build (skinny)
  • GitHub Check: python (2)
  • GitHub Check: python (3)
  • GitHub Check: python (1)
  • GitHub Check: build (dev)

Comment thread mlflow/cli/__init__.py
Comment on lines +553 to +563
if enable_workspaces:
os.environ[MLFLOW_ENABLE_WORKSPACES.name] = "true"
if workspace_store_uri:
os.environ[MLFLOW_WORKSPACE_URI.name] = workspace_store_uri
elif workspace_store_uri:
click.echo(
"Ignoring --workspace-store-uri because workspaces are not enabled. "
"Use --enable-workspaces to activate workspace mode.",
err=True,
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Clarify behavior when workspace URI is provided but workspaces are disabled

Currently, when enable_workspaces is false and workspace_store_uri is set, you:

  • Emit a user-facing click.echo saying --workspace-store-uri is being ignored.
  • Later still pass workspace_store_uri into initialize_backend_stores (see below).

This can be confusing, since the message implies the URI is unused. If the intention is to only use workspace_store_uri when workspaces are enabled, consider omitting it from initialize_backend_stores in this branch. If it is still used for some internal wiring even with workspaces disabled, the message should be softened to something like “Workspaces are disabled; the workspace store will not be used for workspace routing” rather than “ignoring”.

🤖 Prompt for AI Agents
In mlflow/cli/__init__.py around lines 553 to 563, the CLI echoes that
--workspace-store-uri is being ignored when workspaces are disabled but later
still passes workspace_store_uri into initialize_backend_stores, which is
confusing; update the behavior to be consistent by either (A) not passing
workspace_store_uri into initialize_backend_stores when enable_workspaces is
false (so the value is truly ignored), or (B) if the URI is required for
internal wiring even with workspaces off, change the echo message to clearly
state that workspaces are disabled and the workspace store will not be used for
workspace routing while it may still be used internally; implement whichever
option fits the intended design and adjust the conditional logic and message
accordingly.

Comment on lines 138 to 150
def from_proto(cls, proto):
# input: mlflow.protos.model_registry_pb2.RegisteredModel
# returns RegisteredModel entity
# Workspace is intentionally derived from the request context (falling back to the active
# workspace resolver) and therefore is not persisted in the ProtoRegisteredModel.
registered_model = cls(
proto.name,
proto.creation_timestamp,
proto.last_updated_timestamp,
proto.description,
[ModelVersion.from_proto(mvd) for mvd in proto.latest_versions],
workspace=None,
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Fix from_proto argument ordering so workspace is correctly set

In from_proto, the constructor is currently called as:

registered_model = cls(
    proto.name,
    proto.creation_timestamp,
    proto.last_updated_timestamp,
    proto.description,
    [ModelVersion.from_proto(mvd) for mvd in proto.latest_versions],
    workspace=None,
)

Because workspace=None is passed positionally, it is bound to the tags parameter instead of the workspace parameter. Right now this happens to behave the same as omitting tags (they both end up as None), but it’s misleading and fragile if the signature changes.

Prefer an explicit keyword for workspace (and other non-leading args if desired), e.g.:

-        registered_model = cls(
-            proto.name,
-            proto.creation_timestamp,
-            proto.last_updated_timestamp,
-            proto.description,
-            [ModelVersion.from_proto(mvd) for mvd in proto.latest_versions],
-            workspace=None,
-        )
+        registered_model = cls(
+            name=proto.name,
+            creation_timestamp=proto.creation_timestamp,
+            last_updated_timestamp=proto.last_updated_timestamp,
+            description=proto.description,
+            latest_versions=[ModelVersion.from_proto(mvd) for mvd in proto.latest_versions],
+            workspace=None,
+        )

This keeps the intent clear and ensures workspace is wired correctly.

🤖 Prompt for AI Agents
In mlflow/entities/model_registry/registered_model.py around lines 138 to 150,
the call to cls(...) passes workspace=None positionally which binds it to the
tags parameter; change the call to pass workspace explicitly as a keyword (e.g.
workspace=None) so workspace is guaranteed to be set correctly (you may also
pass tags explicitly if needed) and avoid positional ambiguity if the
constructor signature changes.

Comment on lines +105 to +120
#: Specifies the active workspace for client operations.
#: (default: ``None``)
MLFLOW_WORKSPACE = _EnvironmentVariable("MLFLOW_WORKSPACE", str, None)

#: Specifies the workspace provider URI. Falls back to the tracking URI when unset.
#: (default: ``None``)
MLFLOW_WORKSPACE_URI = _EnvironmentVariable("MLFLOW_WORKSPACE_URI", str, None)

#: Specifies the registry URI.
#: (default: ``None``)
MLFLOW_REGISTRY_URI = _EnvironmentVariable("MLFLOW_REGISTRY_URI", str, None)

#: Specifies the workspace provider backend URI.
#: Defaults to the tracking URI when unset.
MLFLOW_WORKSPACE_URI = _EnvironmentVariable("MLFLOW_WORKSPACE_URI", str, None)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Remove duplicate definitions of workspace env vars

MLFLOW_WORKSPACE_URI and MLFLOW_ENABLE_WORKSPACES are each defined twice in this module (lines 109–112 & 117–120, and 450–453 & 1143–1146). While Python will just overwrite the first binding, this duplication is confusing and makes future edits easy to get wrong.

Consider keeping a single definition for each and deleting the earlier or later duplicate, keeping the clearer docstring.

-#: Specifies the workspace provider backend URI.
-#: Defaults to the tracking URI when unset.
-MLFLOW_WORKSPACE_URI = _EnvironmentVariable("MLFLOW_WORKSPACE_URI", str, None)
+#: Specifies the workspace provider backend URI.
+#: Defaults to the tracking URI when unset.
+MLFLOW_WORKSPACE_URI = _EnvironmentVariable("MLFLOW_WORKSPACE_URI", str, None)
@@
-#: Enables workspace-aware multi-tenancy features on the MLflow server.
-#: (default: ``False``)
-MLFLOW_ENABLE_WORKSPACES = _BooleanEnvironmentVariable("MLFLOW_ENABLE_WORKSPACES", False)
+#: Enables workspace-aware (multi-tenant) behavior for MLflow servers and clients.
+#: (default: ``False``)
+MLFLOW_ENABLE_WORKSPACES = _BooleanEnvironmentVariable("MLFLOW_ENABLE_WORKSPACES", False)

Pick one location for each env var (e.g., near the other tracking/registry envs) and delete the other.

Also applies to: 450-453, 1143-1146

🤖 Prompt for AI Agents
In mlflow/environment_variables.py around lines 105-120 (and also review
duplicates at 450-453 and 1143-1146), there are duplicate definitions for
MLFLOW_WORKSPACE_URI and MLFLOW_ENABLE_WORKSPACES; remove the redundant
definitions so each environment variable is defined only once, keep the version
with the clearer docstring placed near other tracking/registry env vars, delete
the earlier/later duplicate occurrence(s), and run a quick search to ensure no
other duplicate bindings remain in the file.

Comment on lines +7 to +8
# When true, users inherit default_permission for the reserved 'default' workspace.
grant_default_workspace_access = false

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

Verify the new configuration is properly wired and documented.

The new grant_default_workspace_access configuration option is added with a conservative false default, which appears appropriate for backward compatibility. However, ensure the following before merging:

  1. Downstream code in the authorization/workspace logic properly reads and honors this configuration flag.
  2. This configuration change is documented (changelog, config guide, migration notes if needed).
  3. The false default aligns with the multi-tenancy security model and doesn't require updates for existing deployments.

Run the following script to verify that the configuration key is being read and used consistently:


🏁 Script executed:

#!/bin/bash
# Description: Verify that grant_default_workspace_access is wired into the authorization logic.

# Search for references to this config key in the codebase
rg -n 'grant_default_workspace_access' --type=py

# Search for where basic_auth.ini is parsed to ensure the new key is handled
rg -n 'basic_auth\.ini' --type=py -A 5 -B 5

Length of output: 2211



---

🏁 Script executed:

```shell
#!/bin/bash
# Verify usage context of grant_default_workspace_access in authorization logic

# Check how grant_default_workspace_access is used in __init__.py
rg -n 'grant_default_workspace_access' mlflow/server/auth/__init__.py -A 3 -B 3

# Check for changelog/documentation files
fd -e md -e rst -e txt . -type f | grep -i -E '(changelog|migration|release|doc|guide)' | head -20

# Search for references to default_permission and workspace in docs
rg -n 'default_permission|default.*workspace' --type=md --type=txt --type=rst 2>/dev/null | head -20

Length of output: 1587


🏁 Script executed:

#!/bin/bash
# Check if there are any CHANGELOG or migration docs
find . -maxdepth 3 -type f \( -name "CHANGELOG*" -o -name "HISTORY*" -o -name "MIGRATION*" \) 2>/dev/null | head -10

Length of output: 14



---

🏁 Script executed:

```shell
#!/bin/bash
# Check CHANGELOG.md for documentation of this feature
rg -n 'grant_default_workspace_access|default workspace|workspace access' CHANGELOG.md -A 2 -B 2

# Also check for any workspace-related changes in recent changelog entries
rg -n 'workspace' CHANGELOG.md | head -30

Length of output: 986


Comment on lines +314 to +405
def _ensure_default_workspace_experiment(self) -> None:
"""
Ensure the default experiment exists in the provider's default workspace when enabled.
"""

provider = self._get_workspace_provider_instance()
default_workspace, supports_default = get_default_workspace_optional(
provider, logger=_logger
)
provider_name = (
type(self._workspace_provider).__name__ if self._workspace_provider else "unknown"
)

if not supports_default:
_logger.warning(
"Workspace provider %s does not expose a default workspace; "
"skipping default experiment bootstrap.",
provider_name,
)
return

if default_workspace is None:
return

with workspace_context.WorkspaceContext(default_workspace.name):
if self.get_experiment_by_name(Experiment.DEFAULT_EXPERIMENT_NAME) is None:
with self.ManagedSessionMaker() as session:
self._create_default_experiment(
session, workspace_override=default_workspace.name
)

def _bootstrap_default_experiment_for_workspace(self, workspace: str) -> None:
"""
Create the default experiment for the specified workspace in an isolated session.
"""
MLflow UI and client code expects a default experiment with ID 0.

with self.ManagedSessionMaker() as bootstrap_session:
self._create_default_experiment(bootstrap_session, workspace_override=workspace)

def _create_default_experiment(self, session, workspace_override: str | None = None):
"""
MLflow UI and client code expects a default experiment with ID 0 in single-tenant mode.
This method uses SQL insert statement to create the default experiment as a hack, since
experiment table uses 'experiment_id' column is a PK and is also set to auto increment.
MySQL and other implementation do not allow value '0' for such cases.
experiment table uses 'experiment_id' column as a PK and is also set to auto increment.
MySQL and other implementations do not allow value '0' for such cases.

ToDo: Identify a less hacky mechanism to create default experiment 0
"""
table = SqlExperiment.__tablename__
creation_time = get_current_time_millis()

if self._workspaces_enabled():
workspace = workspace_override or self._get_active_workspace()
existing = (
session.query(SqlExperiment)
.filter(
SqlExperiment.name == Experiment.DEFAULT_EXPERIMENT_NAME,
SqlExperiment.workspace == workspace,
)
.one_or_none()
)
if existing is not None:
return

experiment = SqlExperiment(
name=Experiment.DEFAULT_EXPERIMENT_NAME,
lifecycle_stage=LifecycleStage.ACTIVE,
artifact_location=None,
creation_time=creation_time,
last_update_time=creation_time,
workspace=workspace,
)
session.add(experiment)
try:
session.flush()
except IntegrityError as exc:
session.rollback()
_logger.debug(
"Default experiment already exists for workspace '%s'; another worker likely "
"created it. Swallowing IntegrityError: %s",
workspace,
exc,
)
return

if not experiment.artifact_location:
experiment.artifact_location = self._get_artifact_location(
experiment.experiment_id, workspace
)
session.flush()
return

table = SqlExperiment.__tablename__

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Default experiment handling per workspace is robust, but depends on migrations for older DBs

The new _ensure_default_workspace_experiment / _bootstrap_default_experiment_for_workspace / _create_default_experiment flow correctly:

  • Honors providers that do not expose a default workspace (warning + early return).
  • Creates a default experiment per workspace only if one doesn’t already exist, handling races via IntegrityError swallow.
  • Keeps the legacy experiment_id=0 creation in non-workspace mode, now with workspace=DEFAULT_WORKSPACE_NAME.

This assumes that existing experiments have their workspace column consistently populated (e.g., via migrations); otherwise, lookups that filter on SqlExperiment.workspace == workspace could fail for legacy rows with NULL workspace. If you expect legacy DBs, ensure the migrations backfill workspace to DEFAULT_WORKSPACE_NAME.

🤖 Prompt for AI Agents
In mlflow/store/tracking/sqlalchemy_store.py around lines 314 to 405, the new
per-workspace default-experiment logic assumes existing DB rows have workspace
populated; on legacy DBs with NULL workspace this will skip matching defaults.
Add a DB migration that backfills NULL SqlExperiment.workspace values to the
DEFAULT_WORKSPACE_NAME for existing experiments, and, as a safety change in this
function, make the existence check tolerant of legacy NULLs by treating NULL
workspace as the default workspace when workspace == DEFAULT_WORKSPACE_NAME
(i.e., match rows where workspace == workspace OR (workspace IS NULL AND
workspace == DEFAULT_WORKSPACE_NAME)); update any related inserts/reads
similarly so legacy rows are considered default until migration runs.

Comment on lines +4915 to 5004
def _runs_query(self, session):
"""Return a SqlRun query scoped to the active workspace."""

query = session.query(SqlRun)
if not self._workspaces_enabled():
return query

workspace = self._get_active_workspace()
return query.join(
SqlExperiment, SqlExperiment.experiment_id == SqlRun.experiment_id
).filter(SqlExperiment.workspace == workspace)

def _trace_query(self, session, for_update_or_delete=False):
"""Return a SqlTraceInfo query scoped to the active workspace when enabled.

Args:
for_update_or_delete: If True, returns a query without joins that can be used
for update/delete operations. Workspace filtering must be
done separately in this case.
"""
query = session.query(SqlTraceInfo)
if for_update_or_delete:
return query

if not self._workspaces_enabled():
return query

workspace = self._get_active_workspace()
return query.join(
SqlExperiment, SqlTraceInfo.experiment_id == SqlExperiment.experiment_id
).filter(SqlExperiment.workspace == workspace)

def _logged_model_query(self, session):
"""Return a SqlLoggedModel query scoped to the active workspace when enabled."""
query = session.query(SqlLoggedModel)
if not self._workspaces_enabled():
return query

workspace = self._get_active_workspace()
# Use a subquery to filter by workspace without adding a join to the main query
# This avoids issues with duplicate rows when additional joins are added for filtering
workspace_experiment_ids = (
session.query(SqlExperiment.experiment_id)
.filter(SqlExperiment.workspace == workspace)
.subquery()
)
return query.filter(
SqlLoggedModel.experiment_id.in_(select(workspace_experiment_ids.c.experiment_id))
)

def _get_logged_model_record(
self, session: sqlalchemy.orm.Session, model_id: str, include_deleted: bool = False
) -> SqlLoggedModel:
query = self._logged_model_query(session).filter(SqlLoggedModel.model_id == model_id)
if not include_deleted:
query = query.filter(SqlLoggedModel.lifecycle_stage != LifecycleStage.DELETED)

logged_model = query.one_or_none()
if logged_model is None:
self._raise_model_not_found(model_id)
return logged_model

def _dataset_query(self, session):
"""Return a SqlEvaluationDataset query scoped to the active workspace when enabled."""

query = session.query(SqlEvaluationDataset)
if not self._workspaces_enabled():
return query
return query.filter(SqlEvaluationDataset.workspace == self._get_active_workspace())

def _dataset_workspace_filter(self):
if not self._workspaces_enabled():
return sql.true()
return SqlEvaluationDataset.workspace == self._get_active_workspace()

def _get_dataset_record(
self, session: sqlalchemy.orm.Session, dataset_id: str
) -> SqlEvaluationDataset:
dataset = (
self._dataset_query(session)
.filter(SqlEvaluationDataset.dataset_id == dataset_id)
.one_or_none()
)
if dataset is None:
raise MlflowException(
f"Evaluation dataset with id '{dataset_id}' not found",
RESOURCE_DOES_NOT_EXIST,
)
return dataset

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion | 🟠 Major

Workspace-scoped query helpers centralize and simplify isolation

The new helpers:

  • _runs_query — scopes SqlRun by joining SqlExperiment on experiment_id and filtering by workspace when enabled.
  • _trace_query — returns either a raw SqlTraceInfo query (for_update_or_delete=True) or a workspace-scoped join with SqlExperiment.
  • _logged_model_query — uses a workspace-scoped experiment-id subquery to avoid duplicate rows when additional joins are added.
  • _dataset_query / _dataset_workspace_filter / _get_dataset_record — encapsulate all evaluation dataset workspace filtering.

These abstractions are well-designed and are reused appropriately throughout the file, making future workspace changes easier.

🤖 Prompt for AI Agents
In mlflow/store/tracking/sqlalchemy_store.py around lines 4915 to 5004, the new
workspace-scoped query helper methods (_runs_query, _trace_query,
_logged_model_query, _dataset_query, _dataset_workspace_filter,
_get_dataset_record) are implemented and reviewed okay; no functional code
changes required—ensure all call sites use these helpers consistently and
add/update unit/integration tests covering workspace-enabled and
workspace-disabled paths to validate behavior.

Comment on lines +121 to +123
assert "scorer_permissions" in tables
assert "registered_model_permissions" in tables
assert version[0] == "8606fa83a998"
assert version[0] == "2ed73881770d"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

Verify completeness of upgrade assertions and Alembic version.

Two concerns:

  1. The test verifies scorer_permissions is added during upgrade (line 121), but doesn't verify workspace_permissions is added, even though the other two tests expect this table to exist. If the upgrade process adds workspace_permissions, it should be asserted here as well.

  2. The Alembic version changed from "8606fa83a998" to "2ed73881770d" - please verify this matches the actual migration version that adds workspace support.

Apply this diff to add the missing assertion:

     assert "alembic_version_auth" in tables
     assert "users" in tables
     assert "experiment_permissions" in tables
     assert "scorer_permissions" in tables
     assert "registered_model_permissions" in tables
+    assert "workspace_permissions" in tables
     assert version[0] == "2ed73881770d"
     assert user == ("testuser", 1)

Run the following script to verify the Alembic version corresponds to the workspace migration:


🏁 Script executed:

#!/bin/bash
# Description: Verify the Alembic version in the migration files

# Search for the migration file with version 2ed73881770d
fd -e py . | xargs rg -l "2ed73881770d"

# Show the migration file content to verify it includes workspace_permissions
fd -e py . | xargs rg -A 5 -B 5 "2ed73881770d"

Length of output: 2373


Let me examine the actual migration file to verify what it creates:

[/run_scripts]


🏁 Script executed:

#!/bin/bash
# Display the full migration file to see what it creates
cat mlflow/server/auth/db/migrations/versions/2ed73881770d_workspace_permissions.py

Length of output: 3744


Now let me verify what the test actually does - specifically whether it upgrades all the way to version 2ed73881770d:

[/run_scripts]


🏁 Script executed:

#!/bin/bash
# Show the test_upgrade_from_legacy_database function to see what migrations it applies
rg -A 70 "def test_upgrade_from_legacy_database" tests/server/auth/db/test_cli.py

Length of output: 2390


Let me verify that scorer_permissions is created in an earlier migration to confirm the pattern:

[/run_scripts]


🏁 Script executed:

#!/bin/bash
# Find the migration that creates scorer_permissions
fd -e py . mlflow/server/auth/db/migrations/versions/ | xargs rg -l "scorer_permissions" | head -5

Length of output: 80


[/run_scripts]


🏁 Script executed:

#!/bin/bash
# Verify the scorer_permissions migration creates the table
cat mlflow/server/auth/db/migrations/versions/0965eb92f5f0_add_scorer_permissions.py

Length of output: 996


Add missing assertion for workspace_permissions table.

The test upgrades to Alembic version 2ed73881770d, which creates both scorer_permissions (in migration 0965eb92f5f0) and workspace_permissions (in migration 2ed73881770d). The test currently asserts scorer_permissions exists but omits the assertion for workspace_permissions. For consistency and completeness, add the missing assertion:

     assert "alembic_version_auth" in tables
     assert "users" in tables
     assert "experiment_permissions" in tables
     assert "scorer_permissions" in tables
     assert "registered_model_permissions" in tables
+    assert "workspace_permissions" in tables
     assert version[0] == "2ed73881770d"
     assert user == ("testuser", 1)

The Alembic version "2ed73881770d" is correct.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
assert "scorer_permissions" in tables
assert "registered_model_permissions" in tables
assert version[0] == "8606fa83a998"
assert version[0] == "2ed73881770d"
assert "scorer_permissions" in tables
assert "registered_model_permissions" in tables
assert "workspace_permissions" in tables
assert version[0] == "2ed73881770d"
🤖 Prompt for AI Agents
In tests/server/auth/db/test_cli.py around lines 121 to 123, the test asserts
presence of "scorer_permissions" but omits asserting the newly created
"workspace_permissions" table for Alembic version 2ed73881770d; add an assertion
that "workspace_permissions" is in the tables collection (placed alongside the
existing scorer_permissions assertion) so the test verifies both permissions
tables exist while leaving the existing version[0] equality check unchanged.

Comment on lines +338 to +345
def _build_endpoint_case_id(handler_name: str, path: str) -> str:
"""
Generate a stable pytest param id by combining the handler name with a sanitized
version of the route path (non-word characters converted to underscores).
"""
sanitized_path = re.sub(r"[\\W]+", "_", path.strip("/"))
return f"{handler_name}__{sanitized_path or 'root'}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Regex in _build_endpoint_case_id does not match its documented behavior

The docstring says “non-word characters converted to underscores”, but the pattern r"[\\W]+" only replaces backslashes and the literal character 'W', not all non-word characters. This is at odds with the intent and could yield odd-looking or less-stable test IDs.

You likely meant to use \W inside the character class:

-    sanitized_path = re.sub(r"[\\W]+", "_", path.strip("/"))
+    sanitized_path = re.sub(r"[\W]+", "_", path.strip("/"))

While this only affects test IDs (not behavior), aligning the regex with the stated intent will avoid confusion and ensure consistent param IDs across paths.

🤖 Prompt for AI Agents
In tests/server/test_handlers.py around lines 338 to 345, the regex r"[\\W]+" in
_build_endpoint_case_id incorrectly matches backslash and 'W' instead of
non-word characters; replace it with a pattern that actually matches non-word
characters (e.g. use r"\W+" in re.sub) so that all non-word characters are
converted to underscores and the generated pytest param ids match the docstring
intent.

Signed-off-by: mprahl <mprahl@users.noreply.github.com>
@mprahl mprahl force-pushed the multi-tenancy-backend-only branch from d6b41b1 to 40b0e90 Compare November 21, 2025 23:05
@mprahl mprahl closed this Nov 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant